Interactive Image Segmentation for Model Adaption and Decision Support

In many fields of Agricultural Management and Agricultural Engineering sophisticated algorithms based on complex environment models are used to generate decision-supporting information from various data sources. However, often these models highly depend on the proper adaption of their complex parameter sets to local ambient conditions and in many cases practitioners are not able to perform this adaption. Therefore a concept is shown here that allows the identification of objects in images and their linkage with meta-data in semi-automatic human-machine interaction. The approach combines the robustness of human experiences against spatially and temporarily local variations and the performance and reproducibility of statistical models. It can also be used as an easy way to adapt models to local ambient conditions, which allows recalibrating them more often, thereby increases stability against changes, iteratively improves them and opens the door for life-long machine learning. The software has been developed within the collaborative research project RemoteFarming.1 in which a remote farming robotic weed control system is being developed. The robotic weed control system will be used for in-row weed treatment in carrots at BBCH-scales 10 to 20 in organic farming. In this field weed control is currently conducted by hand. Within the project's first part RemoteFarming.1a an autonomous field robot – based on the platform BoniRob is being built. It is able to autonomously navigate on the field and has an actuator for mechanical treatment of weeds. Furthermore it uses synchronously triggered cameras and lighting units at different wavelengths which can capture high-contrast images of the plants in a shaded space underneath the robot. The detection/identification of weeds in RemoteFarming.1a is performed in a web-based approach by a remote worker, who marks the weeds in images captured by the robot on the field. Afterwards the mechanical actuator of the robot moves to those positions in the field which have been marked in the respective images and eliminates the weed plants. In the second part RemoteFarming.1b this system will be enriched with weed/crop classifiers and the detection/identification. The user will get a suggestion of possible weeds marked in his view and he can confirm or modify these suggestions before the weed will be treated. The software framework described here allows iteratively generating segmentations for images by human-machine interaction. After a first-shot segmentation the user can add marks in the image and after any added mark the segmentation gets improved. The segmentation is visualized by a semi-transparent ImageMap overlaying the original image. The algorithms that have been tested for performing the segmentation so far are Watershed and Graph-Cuts. During the process any arbitrary segment in the ImageMap – even unconnected regions can be assigned to an object. These objects then can be separated into groups and enriched with additional meta-data. Furthermore the ImageMaps can be grouped into Situations representing different field conditions. The framework's design is flexible with abstraction of front-end and back-end. On the back-end side a server version saves data in a relational database. Alternatively a stand-alone version provides the same functionality using XML to persist data. For the front-end a web-based version can be deployed on servers. Another front-end is implemented as App. This allows using the framework on mobile devices even without Internet connection, saving the gathered data temporarily in XML and persisting into DB once connected. The framework has been used within the collaborative research project RemoteFarming.1 for labeling of crop and weed plants. It allowed generating a sophisticated ground-truth for shape-matching algorithms and weed/crop classifiers. Regions of plants and even overlapping leafs have been marked, grouped to plants and assigned with labels (Species) and meta-data (BBCH-scale etc.). In the on-going project the system will be enriched with statistical models to provide the user improved first-shots for segmentation and plant classification. But geometric analyses of the labelled data collected at project beginning has already served as specific input for vague issues in requirement analysis for the remote farming robotic weed control system that will be developed.